Regularization of Neural Networks using DropConnect
نویسندگان
چکیده
We introduce DropConnect, a generalization of Dropout (Hinton et al., 2012), for regularizing large fully-connected layers within neural networks. When training with Dropout, a randomly selected subset of activations are set to zero within each layer. DropConnect instead sets a randomly selected subset of weights within the network to zero. Each unit thus receives input from a random subset of units in the previous layer. We derive a bound on the generalization performance of both Dropout and DropConnect. We then evaluate DropConnect on a range of datasets, comparing to Dropout, and show state-of-the-art results on several image recognition benchmarks by aggregating multiple DropConnect-trained models.
منابع مشابه
DropAll: Generalization of Two Convolutional Neural Network Regularization Methods
We introduce DropAll, a generalization of DropOut [1] and DropConnect [2], for regularization of fully-connected layers within convolutional neural networks. Applying these methods amounts to subsampling a neural network by dropping units. Training with DropOut, a randomly selected subset of activations are dropped, when training with DropConnect we drop a randomly subsets of weights. With Drop...
متن کاملDropELM: Fast neural network regularization with Dropout and DropConnect
In this paper, we propose an extension of the Extreme Learning Machine algorithm for Single-hidden Layer Feedforward Neural network training that incorporates Dropout and DropConnect regularization in its optimization process. We show that both types of regularization lead to the same solution for the network output weights calculation, which is adopted by the proposed DropELM network. The prop...
متن کاملRegularization for Unsupervised Deep Neural Nets
Unsupervised neural networks, such as restricted Boltzmann machines (RBMs) and deep belief networks (DBNs), are powerful tools for feature selection and pattern recognition tasks. We demonstrate that overfitting occurs in such models just as in deep feedforward neural networks, and discuss possible regularization methods to reduce overfitting. We also propose a “partial” approach to improve the...
متن کاملComparison of Regularization Methods for ImageNet Classification with Deep Convolutional Neural Networks
Large and Deep Convolutional Neural Networks achieve good results in image classification tasks, but they need methods to prevent overfitting. In this paper we compare performance of different regularization techniques on ImageNet Large Scale Visual Recognition Challenge 2013. We show empirically that Dropout works better than DropConnect on ImageNet dataset. © 2013 Published by Elsevier B.V. S...
متن کاملGraphConnect: A Regularization Framework for Neural Networks
Deep neural networks have proved very successful in domains where large training sets are available, but when the number of training samples is small, their performance suffers from overfitting. Prior methods of reducing overfitting such as weight decay, Dropout and DropConnect are data-independent. This paper proposes a new method, GraphConnect, that is data-dependent, and is motivated by the ...
متن کامل